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INTEGRATING MULTIPLE COMMUNICATION MODES 

TECHNICAL FIELD 
This invention generally relates to integrating multiple communication modes. 

BACKGROUND 

5 New communication technologies are becoming available on computers. These 

technologies, primarily based on high speed processors, good connections to high speed data 
transmission on Ethernet and Internet, and fast rendering of graphics, enable new modes of 
personal communication on computers such as voice conversation, voice with video 
conversation, graphic text-based conversation, and electronic mail (e-mail). An example of 
10 voice conversation mode of personal communication on computers is Voice delivered over 

Internet Protocol (VoIP) while an example of graphic text-based conversation mode is Instant 
Messaging (IM). 

VoIP is a term used in Internet Protocol (IP) telephony for a set of facilities for managing 
the delivery of voice information using the Internet Protocol (IP). In general, this means sending 

15 voice information in digital form in discrete packets rather than in the traditional circuit- 
committed protocols of the public switched telephone network (PSTN). VoIP has generally been 
implemented for personal communication on computers using voice conversation mode as a 
replacement for physical telephones. In some examples, the graphical user interface (GUI) 
created as this replacement has been designed with a keypad designed to emulate the keypad of a 

20 physical telephone. Thus, the user must click on different numbers on the GUI keypad to make a 
phone call. 

IM provides the ability to easily determine whether a chosen friend or co-worker is 
connected to the Internet and, if they are, to exchange messages with them. IM differs from 
ordinary e-mail in the immediacy of the message exchange and also makes a continued exchange 
25 simpler than sending e-mail back and forth. These message exchanges are graphic text-based. 
In order for IM to work, both users (who must subscribe to the service) must be online at the 
same time, and the intended recipient must be willing to accept instant messages. If the IM 
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software is set to accept instant messages, the IM software alerts the recipient with a distinctive 
sound, generates a window that indicates that an incoming message has arrived, and allows the 
recipient to accept or reject the incoming message or a window containing the incoming 
message. 

5 Under most conditions, IM is truly "instant." Even during peak Internet usage periods, 

the delay is rarely more than a second or two. Little or no delay makes it possible for two people 
to have a real-time online conversation by sending instant messages back and forth to each other. 

SUMMARY 

10 In general, in one aspect, there is a method that includes receiving from a first person a 

request to converse with a second person using any one of two or more selectable 
communication modes, and in response to the received request, automatically performing an 
action determined by a rule created by the second person. 

Implementations may include one or more other features. The rule is created by the 

15 second person using a user interface on a computing device. The method further includes 

selecting the rule from a set of one or more rules based on a condition statement of the rule. The 
method further includes selecting the rule based on the one of two or more communication 
modes. The method further includes selecting the rule based on an identity of the first person. 
The method further includes selecting the rule based on a current status of the second person. 

20 The method further includes determining an electronic document associated with the first person 
and retrieving the electronic document if the second person indicates a desire to view the 
document. In some cases, the method further includes displaying the electronic document to the 
second person. In other cases, the method further includes retrieving the electronic document 
from an e-mail storage module, wherein the electronic document is an e-mail message. In still 

25 other cases, the method further includes retrieving a calendar of the second person from a 
calendar storage module, wherein the electronic document is the calendar. 

Implementations may include one or more other features. Automatically performing the 
action further includes enabling the first person to leave a message if the current status of the 
second person is that the second person is unavailable to converse. Automatically performing 

30 the action further includes forwarding the request to converse to a third person if a current status 
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of the second person is that the second person is unavailable to converse and the third person is 
available to converse. The one of two or more communication modes includes a voice 
conversation communication mode, in some cases, the voice conversation communication mode 
includes Voice over Internet Protocol (VoIP). The one of two or more communication modes 
5 includes a voice/video conversation communication mode. The one of two or more 

communication modes includes a graphic text-based conversation communications mode, in 
some cases, the graphic text-based conversation communication mode includes Instant 
Messaging. 

In general, in another aspect, there is a system that includes a computing device that 

10 includes a transceiver configured to receive a request to converse with a user of the computing 
device, and an integration module configured to interact with at least two of voice conversation 
software, voice-video conversation software, graphic text-based conversation software, fax 
software, and electronic mail software, and to automatically perform an action determined by a 
rule created by the user based on the received request. 

15 Implementations may include one or more other features. The integration module 

includes a microphone and a speaker. The integration module includes a user interface hook to 
detect when the user is interacting with the computing device. The integration module includes a 
user interface that enables the user to specify the action. The system further includes a network. 
The system further includes a second computing device configured to send the request to 

20 converse. The system further includes a telephone configured to send the request to converse. 
The network includes a switched local area network. The transceiver is further configured to 
receive a request to converse via the switched local area network. The switched local area 
network is configured to connect the computing device to an internet. The switched local area 
network is configured to connect the computing device to an intranet. The switched local area 

25 network is configured to connect to an internet protocol/public switched telephone network 

gateway, in some cases, the network further includes a second switched local area network. In 
such cases, the second computing device can send the request to converse via the second 
switched local area network. In some cases, the network further includes a telephone system and 
a public switched telephone network configured to enable the telephone to send the request to 

30 converse to the computing device. 
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In general, in one aspect, there is a method that includes providing to a first person a 
listing of a set of persons, the listing comprising a name, presence information, and two or more 
communication modes available to communicate with each person. The method also includes 
enabling the first person to select a second person from the set of persons, and 
5 enabling the first person to select a communication mode from the communication modes 
available to communicate with the second person. 

Implementations may include one or more other features. The method further includes 
retrieving one or more first letters of a name of the second person, matching the one or more first 
letters of the name to names of a second set of persons, and presenting the second set of persons 

10 to the first person. In some cases, the method further includes enabling the first person to select 
the second person from the second set of persons. The method further includes enabling the first 
person to communicate with the second person using the selected communication mode by 
interfacing with a computer program. The method further includes enabling the first person to 
communicate with the second person by voice. In some cases, communication by voice uses 

15 Voice over Internet Protocol (VoIP). The method further includes enabling the first person to 

communicate with the second person by voice and video. The method further includes enabling 
the first person to communicate with the second person by text-based conversation. In some 
cases, the text-based conversation includes Instant Messaging. The method further includes 
enabling the first person to communicate with the second person via an e-mail message. The 

20 communication modes include at least two of a voice conversation mode, an e-mail mode, a 
graphic text-based conversation mode, and an voice/video conversation mode. The presence 
information includes an indicator indicating that the second person is (i) logged into a computer, 
(ii) at work but not logged into the computer, or (iii) out of the office. The listing further 
includes status information comprising an indicator indicating that the second person is currently 

25 engaged in conversation with a third person. In some cases, the indicator indicates that the 

second person is currently engaged in conversation using one of a voice conversation mode, a 
voice/video conversation mode, and a graphic text-based conversation mode. The method 
further includes querying a database for information about the set of persons. 

In general, in another aspect, there is a system that includes a user interface module 

30 configured to generate a listing of a set of persons, the listing comprising a name, presence 

information, and communication modes available to communicate with each person. The user 
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interface module is also configured to enable a user to select a person from the set of persons, 
and enable the user to select a communication mode from the communication modes available to 
communicate with the selected person. 

Implementations may include one or more other features. The user interface is further 
5 configured to interface with a computer program providing at least one of the communication 
modes. The user interface is further configured to interface with a computer program providing 
at least a portion of the presence information. The system further includes a database including 
information about the set of persons. The user interface module further includes a user interface 
hook to detect when the user is interacting with the computing device. 

10 In general, in another aspect, there is a communication integration environment that 

includes an integration module configured to interact with at least two of voice conversation 
software, voice- video conversation software, graphic text-based conversation software, fax 
software, and electronic mail software, and to automatically perform an action determined by a 
rule created by the user based on the received request. The environment also includes a user 

15 interface module configured to generate a listing of a set of persons, the listing comprising a 
name, presence information, and communication modes available to communicate with each 
person. The user interface module is also configured to enable a user to select a person from the 
set of persons, and enable the user to select a communication mode from the communication 
modes available to communicate with the selected person. 

20 In general, in another aspect, there is a system that includes a computer device, a user 

interface that is configured to enable a user to interact with a person using one of at least two of 
voice conversation, voice-video conversation, graphic text-based conversation, fax, and 
electronic mail; wherein the interaction includes the following. Creating a rule to cause the 
computer device to automatically perform an action based on a request to converse with the user. 

25 Viewing an automatically generated listing of a set of persons, the listing comprising a name, 

presence information, and communication modes available for the user to communicate with the 
person from the set of persons. Selecting the person from the set of persons. Selecting a 
communication mode from the communication modes available to communicate with the person. 
Retrieving information about a person using an identifying characteristic of the person, and the 

30 identifying characteristic being selected by the user from a display. Communicating with the 
person. 
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In general, in another aspect, there is a method that includes enabling a first person to 
select an identifying characteristic of a second person in a display provided by a first computer 
program, automatically retrieving, using a second computer program, information about the 
second person using the identifying characteristic of the second person and a type of the 
5 characteristic, and enabling the first person to select from the communication modes available to 
contact the second person. 

Implementations may include one or more other features. Retrieving further includes 
determining the type of characteristic. The method further includes determining the 
communication modes available to communicate with the second person based on the identifying 

10 characteristic of the second person. The method further includes determining the communication 
modes available to communicate with the second person based on the type of characteristic. The 
method further includes enabling the first person to communicate with the second person using 
the selected communication mode. In some cases, enabling the first person to communicate with 
the second person further includes interfacing with a third computer program. The method 

15 further includes determining a communication mode identifier associated with the second person 
for at least one of the communication modes available to contact the second person. The 
communication modes include at least two of a voice conversation mode, an e-mail mode, a 
graphic text-based conversation mode, and an voice/video conversation mode. The method 
further includes displaying at least one of a name associated with the second person, presence 

20 information associated with the second person, and status information associated with the second 
person. The presence information includes an indicator indicating that the second person is (i) 
logged into a computer, (ii) at work but not logged into a computer, (iii) out of the office. The 
status information includes an indicator indicating that the second person is currently engaged in 
conversation with another person. In some cases, the indicator indicates that the second person 

25 is currently engaged in conversation using one of voice conversation mode, voice/video 
conversation mode, and graphic text-based conversation mode. 

implementations may include one or more other features. The method further includes 
enabling the first person to communicate with the second person by voice. Communication by 
voice uses Voice over Internet Protocol (VoIP). The method further includes enabling the first 

30 person to communicate with the second person by voice and video. The method further includes 
enabling the first person to communicate with the second person by text-based conversation. In 
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some cases, the text-based conversation uses Instant Messaging. The method further includes 
enabling the first person to communicate with the second person via an e-mail message. 
Enabling the first person to select an identifying characteristic of the second person further 
includes highlighting the identifying characteristic. The identifying characteristic includes a 
5 name of the second person. The identifying characteristic includes a telephone number of the 
second person. The method further includes performing optical character recognition on an 
image of the identifying characteristic. The identifying characteristic includes an image of the 
second person. 

In general, in another aspect, there is a system that includes a retrieval module configured 

10 to automatically retrieve information about a first person using an identifying characteristic of 
the first person, and the identifying characteristic being selected by a second person from a 
display, and a selection module configured to enable the second person to select from 
communication modes available to communicate with the first person. 

Implementations may include one or more other features. The system further includes a 

15 network. The selection module is further configured to interface with a computer program to 

provide the second person with one of the communication modes available to communicate with 
the first person. The system further includes a display module to display at least one of a name 
associated with the first person, presence information associated with the first person, and status 
information associated with the first person. In some cases, the presence information includes an 

20 indicator indicating that the first person is (i) logged into a computer, (ii) at work but not logged 
into a computer, (iii) out of the office. In other cases, the status information includes an indicator 
indicating that the first person is currently engaged in conversation with another person. In these 
cases, the indicator can indicate that the first person is currently engaged in conversation using 
one of voice conversation mode, voice/video conversation mode, and graphic text-based 

25 conversation mode. The selection module includes a user interface hook to detect when the user 
is interacting with the computing device. 

In general, in another aspect, there is a system that includes a computer device and a user 
interface. The user interface is configured to enable a user to interact with a person using one of 
at least two of voice conversation, voice- video conversation, graphic text-based conversation, 

30 fax, and electronic mail. The interaction includes the following. Creating a rule to cause the 

computer device to automatically perform an action based on a request to converse with the user; 
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viewing an automatically generated listing of a set of persons, the listing comprising a name, 
presence information, and communication modes available for the user to communicate with the 
person from the set of persons. Selecting the person from the set of persons. Selecting a 
communication mode from the communication modes available to communicate with the person; 
retrieving information about a person using an identifying characteristic of the person, where the 
identifying characteristic is selected by the user from a display. Communicating with the person. 

In general, in another aspect there is an article comprising a machine-readable medium 
that stores executable instruction signals that cause a machine to perform one or more of the 
methods described above. 

These and other embodiments may have one or more of the following advantages. 

Multiple modes of personal communication on computers are integrated together with a 
contact list and user's preferences to improve the user's communication with others. The 
communication is improved by enabling the user to specify the identity of another person using 
convenient techniques, view presence information about the other person, and then specify the 
desired mode of communication with the other person by simply selecting the desired mode from 
the integrated multiple modes. The communication is also improved by enabling the user to set 
up rules that determine how incoming requests for communication with the user are 
automatically handled. 

In a corporate environment, integrating the multiple modes of personal communication 
on computers enables the user's entire computer-based communication environment to follow 
the user from a computer in an office to another computer in another office. This can allow 
office communication equipment, such as a phone and a desktop computer, to remain at one 
physical location and the employee's communication environment to follow that employee, 
reappearing when the employee logs into another computer in another office. 

The details of one or more examples are set forth in the accompanying drawings and the 
description below. 

DESCRIPTION OF DRAWINGS 
FIG 1 is a view of a communication system environment. 
FIG 2 is a view of a computer. 
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FIG 3 is a view of a user interface to integrate multiple modes of personal 
communication on a computer. 

FIG 3 A is a view of a user interface for voice conversation mode of personal 
communication. 

5 FIG 4 is a view of an Internet web browser with the text of a telephone number 

highlighted. 

FIG 4A is view of a user interface for voice conversation mode of personal 
communication after highlighting a telephone number in the web browser of FIG 4. 

FIG 5 is a view of a user interface of a rules assistant with a rule. 
10 FIG 6 is a view of a user interface for voice conversation mode of personal 

communication enabling a user to bring up documents associated with the caller. 

FIG. 7 is a view of components of communication integration software. 

FIG 8 is a view of another communication system. 

DETAILED DESCRIPTION 

15 Referring to FIG 1, communication system 10 includes voice-enabled computer 14, 

network 16, voice-enabled computer 24, telephone 28, and voice-enabled computer 30. 
Computers 14, 24, and 30 are computing devices that can include, for example, desktop 
computers and/or handheld computers. Computer 14, computer 24, and computer 30 operate 
according to instructions in integration software 32. Integration software 32 integrates multiple 

20 modes of personal communication on each computer and provides an interface to a user 12 using 
a single graphical user interface (GUI). Examples of the multiple modes of personal 
communication can include voice conversation (e.g., VoIP), voice-video conversation, graphic 
text-based conversation (e.g., IM), or other forms of graphic text-based communication such as 
fax or electronic mail (e-mail). User 12 uses voice-enabled computer 14 to communicate with 

25 users 22 and 26 via network 16. User 22 uses voice-enabled computer 24 while user 26 uses 
telephone 28. Computer 14 has access to server 18 that is connected to database 20. User 12 
communicates using one or more of the multiple modes of personal communication on computer 
14. Database 20 stores the communication preferences of user 22 and the identities as well as 
communication addresses of users 22 and 26. Although database 22 is a separate entity in the 
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! 
i 

i illustrated example, database 22 or portions of database 22 can be distributed and/or included on 

a computing device, such as 14, 24, and/or 30. 

In system 10, voice conversation mode of personal communication via computer 24, in 
addition to other modes of personal communication, is controlled by instructions executed by 
5 computer 14 and components of these modes of personal communication are represented by data 
in database 20. Thus, these components can be manipulated, stored, categorized, accessed, and 
moved around network 16 like any other form of data. For example, integration software 32 
enables user 12 to set instructions to forward or store a voice message using the same user 
interface as forwarding an e-mail message. These instructions and the components of the modes 

10 of personal communication are stored in database 20. User 12 may also log into computer 30 
and have access to the same information on computer 30 from database 20 as by logging into 
computer 14. In corporate environments, for example, users can move around between different 
locations providing computers such as computer 14 and computer 30. These different locations 
can include, for example, user 12's permanent office, user 12's colleague's office, a conference 

15 room over wireless LAN connection, an airport lounge with a Virtual Private Network (VPN) 

connection to user 12's employer, and user 12's home office with a VPN connection to server 18. 
No matter where user 12 is logged in using computer 14, 24, or 30, accessibility and control over 
these multiple modes of personal communication on computers are available to user 12. 

If user 12 is logged in at computers 14 and 30 concurrently and user 22 sends a request to 

20 converse to user 12, then computers 14 and 30 can both indicate the arrival of the request for 
conversation. If user 12 does not answer at either computer 14 or 30, then, based on the 
instructions of user 12, integration software 32 can deliver a message from user 22 to a mailbox 
stored in database 20, accessible at either computer 14 or 30. 

When either user 22 or user 26 sends a request for conversation to user 12 and user 12 is 

25 logged in on computer 14, integration software 32 on computer 14 detects the identity of either 
user 22 or user 26. A request to converse can include, for example, an incoming phone call using 
VoIP or an incoming instant message. When the request is for voice conversation, integration 
software 32 on computer 14 detects the identity based on, for example, the automatic number 
identification (ANI) information provided with the telephone call by matching the ANI with the 

30 requester's identity in database 20. Other requests for conversation can directly contain the 
identity of the requesting user. 
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When either user 22 or user 26 sends a request for conversation to user 12 and user 12 is 
logged in on computer 14, computer 14 can use the identity of the requesting user to search 
different contact and address lists in database 20 and display as much information about the 
incoming call and user 22 or 26 as computer 14 is able to retrieve from database 20. Such 
5 information can include, for example, user 22's or 26's organization, title, and/or photo. Based 
on a simple action by user 12 using a GUI, integration software 32 on computer 14 can also 
retrieve associated documents, such as recent e-mail correspondence to or from user 22 or 26, 
stored in database 20 so that this recent correspondence is available during conversation. In 
another example, integration software 32 on computer 14 can also retrieve calendar information 

10 for user 22 or 26 in database 20 so that the availability of user 22 or 26 is available during the 

conversation based on a simple action by user 12. This GUI is described in greater detail below. 

When user 12 goes on vacation, user 12 can use integration software 32 to set a vacation 
message (once) in a computer desktop environment setting to store the message in database 20, 
and integration software 32 can configure all of the communication modes available to user 12 

15 based on that single vacation message. 

Using integration software 32, user 12 can manage communication distribution groups 
(e.g., groups in the Global Address List) in one place and communicate with these groups using 
any of the multiple modes of personal communication. 

While preparing to send an e-mail to a list including users 22 and 26, user 12 decides to 

20 include voice annotation associated with the e-mail. Because integration software 32 coordinates 
with the various modes of personal communication, user 12 can include the voice annotation in 
the e-mail, or user 12 can initiate an immediate synchronous phone connection to all users in the 
list including users 22 and 26. 

A set of rules defined by user 12 does a portion of the integration of multiple modes of 

25 personal communication on a computer for user 12. User 12 uses a rules assistant in the form of 
a GUI to specify these rules, as discussed in greater detail below. Using this rules assistant, user 
12 can specify actions computer 14 should take when certain events such as an incoming request 
to converse, from user 22 to user 12, arrives at computer 14. The condition statements of these 
rules specify the parameters of events. These parameters include, for example, the identity of 

30 user 22 (the sender), the mode of conversation used, a date and time range, and the current status 
of user 12 (the recipient). In an illustrative example, conditions of a particular rule include 
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receipt of a voice call and user 12 is out of the office. In defining a rule for this example, user 12 
can specify actions such as send the voice call to voice mail, disconnect the voice call, or 
forward the voice call to some other user. 

Referring to FIG 2, computer 14 includes monitor 58, processor 60, and storage medium 
5 62. Processor 60 executes instructions stored in storage medium 62 while monitor 58 displays 
graphical output from processor 60. Storage medium 62 stores operating system 64, network 
software 66, and applications. Applications include integration software 32 that integrates 
multiple communication modes for user 12 on computer 14. Applications also include Internet 
browser software 70, calendar software 72, e-mail software 74, VoIP software 76, IM software 

10 80, and voice/video communication software 82. Computer 14 also includes one or more 
speakers 84, microphone 86, and camera 88. VoIP software 76 uses speaker 84 to transmit 
speech from user 22 or user 26 to user 12 while VoIP software 76 uses microphone 86 to record 
speech from user 12. Voice/video communication software 82 uses speaker 84 and microphone 
86 in a similar way as VoIP software and also uses camera 88 to record video of user 12. 

15 One example of Internet browser software 70 is available as Internet Explorer® from 

Microsoft Corporation of Redmond, WA. One example of calendar software 72 is available as 
Outlook® software from Microsoft Corporation of Redmond, WA. One example of e-mail 
software 74 is also available as Outlook® software from Microsoft Corporation of Redmond, 
WA. One example of VoIP software 76 is available as WinRTP® as source code from 

20 Vovida.org and another example as Telephony Application Programmer's Interface® (TAPI) 
software from Microsoft Corporation of Redmond, WA. One example of voice/video 
communication software 82 is available as Eyeball Chat® from Eyeball Networks of Vancouver, 
BC, Canada. 

Integration software 32 incorporates a rules assistant function to enable user 12 to setup 
25 rules determining how incoming communication is handled. Integration software 32 also 

interacts with Application Programmer's Interfaces (APIs) in software 70, 72, 74, 76, 80, and 82 
to integrate multiple communication modes for user 12 on computer 14. This integration 
includes reading electronic information stored by software 70, 72, 74, 76, 80, and 82 and 
launching the execution of components of software 70, 72, 74, 76, 80, and 82. Examples of this 
30 integration are discussed in greater detail below. 
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Referring to FIG 3, integration software 32 provides a Graphical User Interface (GUI) 
102 that provides a person-centric interface that enables user 12 to contact user 28 from a list of 
known persons using multiple communication modes via computer 14. In the illustrated 
example, these modes include a voice conversation mode, facilitated by VoEP software 76, and 
5 graphic text-based conversation mode, facilitated by IM software 80, as well as e-mail 

communication mode, facilitated by e-mail software 74. GUI 102 includes area 112 with 
available communication modes. Area 112 includes button 128 (to specify VoIP 
communication), button 130 (to specify e-mail), and button 132 (to specify IM communication). 
GUI 102 also presents the name 104 of user 12 and a list of contacts 106. For each 

10 contact in list 106 in GUI 102, when user 12 selects that particular contact, the buttons 112 
representing communication modes that are available for communication with that contact 
selected from list 106 are activated, while buttons 112 representing communication modes that 
are not available for communication with that contact selected from list 106 are deactivated. 
Thus, GUI 102 provides user 12, in a single interface, with a list of communication modes that 

15 are available to communicate with a selected contact from list 106, even though applications 

providing that mode of communication are independent of GUI 102. In GUI 102, icons 114 and 
1 1 5 show different IM providers that are available for a contact. Icon 1 1 8 shows whether the 
contact is currently using a phone, and name 120 of the contact is color coded to reflect calendar 
information of the contact in list 106. In an example that uses Outlook® as calendar software 

20 72, GUI 102 can perform the color-coding of the names according to the Outlook® status of an 
individual and uses the same color-coding as found in Outlook®. In that case, blue means busy, 
purple means out of office, and light blue means tentative. Area 122 provides one way for user 
12 to specify the identity of user 22 to contact by typing in the name or telephone number of user 
22. Areas 124 and 126 present auxiliary information to user 22 such as weather, stock market 

25 conditions, and news. 

To communicate with user 22 or user 26, user 12 first identifies user 22 or user 26 using 
GUI 102. User 12 does this by clicking on a name in list 106 or typing in an identifier in area 
122. This identifier enables user 12 to identify a person not in list 106 but who is in database 20. 
This identifier may be a name of a person in one or more databases 20, where the name can be a 

30 nickname or user name, or a telephone number in database 20 with a match to a person. This 

identifier can also be a string of alphabetic characters of the beginning of the name of a person in 
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database 20. In general, if the identifier does not uniquely identify a person in database 20, a 
sublist of persons is presented to the user and the user can select a person from this sublist. 
Otherwise, if the person is uniquely identified, integration software 32 initiates the 
communication process with the identified person. Before identifying which person to 
5 communicate with, user 12 can check the presence information of persons in list 106. 

The icons 114 and 1 1 5 of a person standing or running indicate support for different IM 
services managed by IM software 80. For example, the running person icon is representative of 
the AOL® IM service, the person standing is a Fidelity One contact. Other examples include 
icons of a Y for Yahoo®, a flower for ICQ®, and a pawn looking icon for MSN Messenger®. 

10 Icons 114 and 1 1 5 are shaded, representing the state of the user in relation of the user's IM 

usage. Green means that the user is currently engaged in an IM conversation, red means the user 
is away from the office, blue means do not disturb the person, gray means the user is at work but 
not logged into the computer, and white means the user has no presence information. The phone 
icon 118 means that Brian Falvey is currently speaking on a VoIP phone. The shading of name 

15 120 indicates the office calendar status of user 12. GUI 102 interacts with calendar software 72 
to determine the calendar status of user 12. For example, Outlook® software allows user 102 to 
set his or her current status at any given time as free, tentative, busy, or out of the office. 

Once user 12 has identified a person to communicate with, user 12 can choose a mode of 
communication in area 112. If user 12 wishes to converse with user 22, user 12 can click on 

20 button 128 for voice conversation mode of personal communication using VoIP software 76 or 
click on button 132 for graphic text-based conversation in the form of IM using IM software 80. 
Otherwise, user 12 can initiate graphic text-based communication in the form of e-mail using e- 
mail software 74 by clicking on button 130. In other examples, other buttons enable user 12 to 
choose other modes of personal communication not illustrated, such as voice/video conversation 

25 using voice/video software 82. 

Referring to FIG. 3 A, integration software 32 responds to user 12 clicking on button 128 
by retrieving a telephone number and other information for a contact specified by user 12 in 
database 20. Integration software 32 then launches the execution of VoIP software 76 that 
presents GUI 200 to user 12. GUI 200 shows information about the VoIP call. Information for 

30 GUI 200 is gathered from database 20. GUI 200 includes name 202 of user 22 or user 26 with 
whom user 12 wishes to communicate. GUI 200 also includes telephone number 204 of user 22 
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or user 26. Additional information about user 22 or user 26 includes title 206 and company 208. 
Because integration software 32 integrates multiples modes of communication, integration 
software 32 can display other information about user 22 obtained from database 20 or through 
other applications (e.g., 72, 74,76,80, and 82) that have access to information about user 22. 
5 Status 210 indicates the status of the VoIP call. This status can be, for example, "Dialing" or 

"Connected". Button 212 enables user 12 to end the conversation. Control 214 enables user 12 
to change the volume for microphone 86 while control 216 enables user 12 to change the volume 
control for computer speaker 84. 

Referring to FIG 4, Internet browser software 70 provides GUI 300. Integration software 

10 32, running in the background of Internet browser software 70 or some other software with a text 
GUI, coordinates with Internet browser software 70 and the other software to enable user 12 to 
communicate with user 22 or user 26 regardless of the software generating a display. For 
example, user 12 selects text 302 of a telephone number in GUI 300 generated by software 70. 
After selecting text 302, user 12 hits a "hot" key combination, such as CTRL D, that is 

15 programmed using the rules assistant. Integration software 32, running in the background of 
Internet browser software 70, detects the event of the "hot" key combination and reads the 
selected text. Integration software 32 determines that the type of the selected text is a number 
and determines that, because the format of the number is 3 digits, hyphen, 3 digits, hyphen, four 
digits, the number is a telephone number. Integration software 32 launches the execution of VoIP 

20 software 76 that brings up GUI 410, as shown in FIG 4A, and passes the telephone number, 

converted from the selected text, to VoIP software 76. GUI 410 enables user 12 to call via VoIP 
the number selected in text 302. The same number appears in box 412. 

In the previous example, because the chosen text was determined to be a phone number, 
integration software 32 automatically selected the communication mode of voice conversation. 

25 The selected text is not, however, always in the form of a phone number. In other examples, this 
selected text can be another identifying characteristic of a contact in database 20. In addition to a 
phone number, other identifying characteristics can include, for example ASCII characters 
representing the name of a contact or an image associated with a contact, such as scanned in text 
or a graphical image. 

30 In one example where the identify characteristic is ASCII characters representing the 

name of a contact, user 12 selects text containing the name of a contact in database 20 and hits 
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the "hot" key combination. Integration software 32, running in the background of Internet 
browser software 70 or other software, detects the event of the "hot" key combination and reads 
the selected text. Integration software 32 determines that the selected text includes alphabetic 
text, that the type of the text is a name, and that the name is an identifying characteristic of a 
5 contact in database 20. Integration software 32 matches the name to the contact in database 20. 
Integration software 32 provides GUI 102 to user 12 with the contact selected and enables user 
12 to select the mode of communication in area 112. In the case where user 12 selects voice 
conversation mode, using button 128, integration software 32 launches the execution of VoIP 
software 76 and passes the telephone number for the contact, retrieved from database 20, to VoIP 
10 software 76. 

In another example where the text is in a scanned image, user 12 selects the applicable 
portion of the image in Internet browser software 70 or some other software and hits the correct 
"hot" key combination. This graphic image contains text of an identifying characteristic such as 
a name of a contact in database 20. If the text is numeric, then the text represents a telephone 

15 number that can be matched to a name of the contact in database 20. If the text is alphabetic, 
then the type of text is a name that is possibly known as a contact in database 20. Integration 
software 32 performs optical character recognition to convert the image to ASCII text and to 
determine the identifying characteristic such as a name or telephone number. Integration 
software 32 compares the ASCII identifying characteristic with data in database 20 to determine 

20 a match. Integration software 32 provides GUI 102 to user 12 with the contact selected and 
enables user 12 to select the mode of personal communication in area 112 as in the previous 
examples. 

In another example, user 12 selects a graphic image in Internet browser software 70 or 
some other software and hits the correct "hot" key combination. This graphic image contains an 

25 identifying characteristic, such as a image of the face of a contact in database 20. After 
determining that the graphic image does not represent numeric or alphabetic characters, 
integration software 32 determines that the type of image is an image of the face of a contact in 
database 20. Integration software 32 , performs image pattern recognition to identify the contact 
in database 20. Integration software 32 , provides GUI 102 to user 12 with the contact selected 

30 and enables user 12 to select the mode of communication in area 1 12 as in the previous 

examples. Software for such facial image pattern recognition is available, for example, from 
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SeeStorm US A of Encinitas, CA. Referring to FIG 5, integration software 32 includes GUI 500 
to enable user 12 to bring up all e-mails from user 22 when user 22 requests a voice conversation 
with user 12 and user 12 is in the office. In operation, upon receiving the request to converse 
from user 22, GUI 500 appears and user 22 can indicate a desire to view the e-mails by selecting 
5 menu item 502. When user 12 selects menu item 502, integration software 32 brings up stored e- 
mails from user 22 for assisting user 12 in conversation with user 22. Integration software 32 
can retrieve these stored e-mails from database 20 and/or e-mail software 74. Integration 
software 32 displays the stored e-mails using e-mail software 74. 

In another example, a similar GUI enables user 22 to bring up calendar information for 
10 user 22 so that the availability of user 22 is visible to user 12 during the conversation. 

Integration software 32 can retrieve this calendar information from database 20 and/or calendar 
software 72. Integration software 32 displays this calendar information through interaction with 
calendar software 72. 

Referring to FIG 6, GUI 540 shows an example of a rule created by user 12 using the 

15 rules assistant. In the illustrated example, GUI 540 enables user 12 to program a rule when 
another user (e.g., user 22) wants to initiate conversation with user 12 via voice. In this case, 
user 12 names the rule in box 542 as "In Meeting". User 12 can set the conditions for the rule in 
area 544 and the actions for the rule in area 546. In area 544, user 12 can set the condition for 
the status of user 12 by clicking on box 545 and selecting from pull down menu 548. In this 

20 case, user 12 clicks on box 546 and selects the status "In a Meeting". Other status conditions in 
pull down menu 548 include "On Vacation", "Sick", "At Work On Computer", and "At Work but 
Not On Computer". User 12 also can set the date and time range when the rule applies. The date 
range is from date in box 550 to date in box 552. User 12 can specify the condition for the name 
of user 22 who requests conversation with user 12 in box 554. Lastly, for this example, GUI 540 

25 provides four possible actions in area 546 to be automatically performed by computer 14 if user 
22 initiates a voice call to user 12 and the conditions specified in area 544 are true. By selecting 
box 556, user 12 specifies that computer 14 sends the call directly to voice mail. By selecting 
box 558, user 12 specifies that computer 14 disconnects the call. By selecting box 560, user 12 
specifies that computer 14 forwards the call to wherever user 12 is currently located. By 

30 selecting box 562, user 12 specifies that computer 14 forwards the call to another user listed in 
database 20. 
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Referring to FIG 7, Fidelity One software 602 is an example of integration software 32 
for the Windows® operating system. Fidelity One software 602 interacts with WinRTP software 
606 and TAPI Control software 608 to provide VoIP services to user 12. The Real-Time 
Transport Protocol (RTP) is an Internet protocol standard that specifies a way for programs to 
5 manage the real-time transmission of multimedia data over either unicast or multicast network 
services. WinRTP is a Component Object Model (COM) component that can originate RTP 
media from a microphone and terminate RTP media on a speaker. That is, WinRTP encodes and 
decodes RTP packets. WinRTP consists of two independent parts. One part has the ability to 
capture the user's voice using microphone 86, encode the user's voice, and send the voice as an 

10 RTP stream to a configurable destination. The other part listens for an RTP stream from the 

network, extracts the audio from the RTP stream, and plays the extracted audio using speaker 84. 
COM is the fundamental "object model" on which ActiveX Controls and OLE are built. COM 
allows an object to expose its functionality to other components and to host applications. COM 
defines both how the object exposes itself and how this exposure works across processes and 

15 across networks. COM also defines the object's life cycle. This makes WinRTP easy to use 
using any programming language like C, C++, or Java (using J/Direct). 

TAPI (Telephony Application Program Interface) software 608 is a standard program 
interface that lets a computer communicate over telephones or video phones to people or phone- 
connected resources elsewhere in the world. 

20 Integration software 32 also links to Easy Message® (EM) Control software 604 to use 

different IM services. EM is a unified instant messenger that provides support for EM providers 
such as ICQ®, MSN®, Yahoo®, and AOL® at the same time. 

Keyboard hook 610 is used in trapping keyboard messages to achieve high speed dialing. 
The idle User Interface (UI) hook 612 is used to detect when a user is at their keyboard by 

25 trapping all mouse and keyboard messages in Windows® to set presence status for IM. 

FIG. 8 shows communication system 732. System 732 includes a network 734 which can 
be the Internet or a corporate intranet. System 732 also includes a Public Switched Telephone 
Network (PSTN) 736. PSTN 736 can include, for example, a portion of the worlds collection of 
interconnected voice-oriented public telephone networks, both commercial and government- 

30 owned. Voice-enabled computers 14, 738, and 740 are connected to a switched Local Area 
Network (LAN) 742. Server 18 is also connected to LAN 742. Voice-enabled computer 24 
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connects to network 734 via LAN 744. LAN 744 also connects computer 24 to PSTN 736 via 
IP/PSTN Gateway 746. LAN 742 connects computers 14, 738, and 740 to PSTN 736 via 
IP/PSTN Gateway 748. Telephone 28 connects to PSTN 736 via private branch exchange (PBX) 
750. PBX 750 is a telephone system within an enterprise that switches calls between enterprise 
users on local lines while allowing users to share a certain number of external phone lines. 
Telephone 28 can also be connected to network 734 via PBX 50 and IP/PSTN Gateway 752. 

System 732 allows flexibility in handling voice and data communication between users 
12, 22, and 26. For conversation modes of communication, system 732 separates data of the 
conversations into Internet Protocol (IP) packets and sends those packets via networks such as 
LANs and the Internet. For conversations using continuous streams of information, such as 
voice conversation and voice/video conversation modes of communication, PSTN 736 can be 
used in combination with network 734 via gateways 746, 748, and 752. For example, one 
communication path for voice conversation mode of communication between users on computers 
14 and 24 is from computer 14 to LAN 742 to IP/PSTN Gateway 748 to PSTN 736 to IP/PSTN 
Gateway 744 to LAN 744 to computer 24. Another communication path for voice conversation 
mode between users on computer 14 and telephone 28 can be from computer 14 to LAN 742 to 
IP/PSTN 748 to PSTN 736 to PBX 750 to telephone 28. For modes of communication other 
than voice conversation and the voice component of voice/video conversation, system 732 
transmits data via LANs 742 and 744 or network 734, for example, without going through PSTN 
736. 

A number of embodiments of the invention have been described. Nevertheless, it will be 
understood that various modifications may be made without departing from the spirit and scope 
of the invention. Accordingly, other embodiments are within the scope of the following claims. 
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